Search for: All records

The pursuit of robustness has recently been a popular topic in reinforcement learning (RL) research, yet the existing methods generally suffer from computation issues that obstruct their real-world implementation. In this paper, we consider MDPs with low-rank structures, where the transition kernel can be written as a linear product of feature map and factors. We introduce *duple perturbation* robustness, i.e. perturbation on both the feature map and the factors, via a novel characterization of (𝜉,𝜂) -ambiguity sets featuring computational efficiency. Our novel low-rank robust MDP formulation is compatible with the low-rank function representation view, and therefore, is naturally applicable to practical RL problems with large or even continuous state-action spaces. Meanwhile, it also gives rise to a provably efficient and practical algorithm with theoretical convergence rate guarantee. Lastly, the robustness of our proposed approach is justified by numerical experiments, including classical control tasks with continuous state-action spaces. 

Off-policy evaluation (OPE) is one of the most fundamental problems in reinforcement learning (RL) to estimate the expected long-term payoff of a given target policy with \emph{only} experiences from another behavior policy that is potentially unknown. The distribution correction estimation (DICE) family of estimators have advanced the state of the art in OPE by breaking the \emph{curse of horizon}. However, the major bottleneck of applying DICE estimators lies in the difficulty of solving the saddle-point optimization involved, especially with neural network implementations. In this paper, we tackle this challenge by establishing a \emph{linear representation} of value function and stationary distribution correction ratio, \emph{i.e.}, primal and dual variables in the DICE framework, using the spectral decomposition of the transition operator. Such primal-dual representation not only bypasses the non-convex non-concave optimization in vanilla DICE, therefore enabling an computational efficient algorithm, but also paves the way for more efficient utilization of historical data. We highlight that our algorithm, \textbf{SpectralDICE}, is the first to leverage the linear representation of primal-dual variables that is both computation and sample efficient, the performance of which is supported by a rigorous theoretical sample complexity guarantee and a thorough empirical evaluation on various benchmarks. 

Balancing relative expression of pathway genes to minimize flux bottlenecks and metabolic burden is one of the key challenges in metabolic engineering. This is especially relevant for iterative pathways, such as reverse β-oxidation (rBOX) pathway, which require control of flux partition at multiple nodes to achieve efficient syn thesis of target products. Here, we develop a plasmid-based inducible system for orthogonal control of gene expression (referred to as the TriO system) and demonstrate its utility in the rBOX pathway. Leveraging effortless construction of TriO vectors in a plug-and-play manner, we simultaneously explored the solution space for enzyme choice and relative expression levels. Remarkably, varying individual expression levels led to substantial change in product specificity ranging from no production to optimal performance of about 90% of the theoretical yield of the desired products. We obtained titers of 6.3 g/L butyrate, 2.2 g/L butanol and 4.0 g/L hexanoate from glycerol in E. coli, which exceed the best titers previously reported using equivalent enzyme combinations. Since a similar system behavior was observed with alternative termination routes and higher-order iterations, we envision our approach to be broadly applicable to other iterative pathways besides the rBOX. Considering that high throughput, automated strain construction using combinatorial promoter and RBS libraries remain out of reach for many researchers, especially in academia, tools like the TriO system could democratize the testing and evaluation of pathway designs by reducing cost, time and infrastructure requirements. 

Abstract Grapevine (Vitis vinifera) is an economically important fruit crop worldwide. The widely cultivated grapevine is susceptible to powdery mildew caused by Erysiphe necator. In this study, we used CRISPR-Cas9 to simultaneously knock out VviWRKY10 and VviWRKY30 encoding two transcription factors reported to be implicated in defense regulation. We generated 53 wrky10 single mutant transgenic plants and 15 wrky10 wrky30 double mutant transgenic plants. In a 2-yr field evaluation of powdery mildew resistance, the wrky10 mutants showed strong resistance, while the wrky10 wrky30 double mutants showed moderate resistance. Further analyses revealed that salicylic acid (SA) and reactive oxygen species contents in the leaves of wrky10 and wrky10 wrky30 were substantially increased, as was the ethylene (ET) content in the leaves of wrky10. The results from dual luciferase reporter assays, electrophoretic mobility shift assays and chromatin immunoprecipitation (ChIP) assays demonstrated that VviWRKY10 could directly bind to the W-boxes in the promoter of SA-related defense genes and inhibit their transcription, supporting its role as a negative regulator of SA-dependent defense. By contrast, VviWRKY30 could directly bind to the W-boxes in the promoter of ET-related defense genes and promote their transcription, playing a positive role in ET production and ET-dependent defense. Moreover, VviWRKY10 and VviWRKY30 can bind to each other's promoters and mutually inhibit each other's transcription. Taken together, our results reveal a complex mechanism of regulation by VviWRKY10 and VviWRKY30 for activation of measured and balanced defense responses against powdery mildew in grapevine.